Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REFACTOR] store webhook event in database #29145

Merged
merged 10 commits into from
Mar 7, 2024

Conversation

oliverpool
Copy link
Contributor

@oliverpool oliverpool commented Feb 12, 2024

Refactor the webhook logic, to have the type-dependent processing happen only in one place.


Current webhook flow

  1. An event happens
  2. It is pre-processed (depending on the webhook type) and its body is added to a task queue
  3. When the task is processed, some more logic (depending on the webhook type as well) is applied to make an HTTP request

This means that webhook-type dependant logic is needed in step 2 and 3. This is cumbersome and brittle to maintain.

Updated webhook flow with this PR:

  1. An event happens
  2. It is stored as-is and added to a task queue
  3. When the task is processed, the event is processed (depending on the webhook type) to make an HTTP request

So the only webhook-type dependent logic happens in one place (step 3) which should be much more robust.

Consequences of the refactor

  • the raw event must be stored in the hooktask (until now, the pre-processed body was stored)
  • to ensure that previous hooktasks are correctly sent, a payload_version is added (version 1: the body has already been pre-process / version 2: the body is the raw event)

So future webhook additions will only have to deal with creating an http.Request based on the raw event (no need to adjust the code in multiple places, like currently).

Moreover since this processing happens when fetching from the task queue, it ensures that the queuing of new events (upon a git push for instance) does not get slowed down by a slow webhook.

As a concrete example, the PR #19307 for custom webhooks, should be substantially smaller:

  • no need to change services/webhook/deliver.go
  • minimal change in services/webhook/webhook.go (add the new webhook to the map)
  • no need to change all the individual webhook files (since with this refactor the *webhook_model.Webhook is provided as argument)

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 12, 2024
@pull-request-size pull-request-size bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Feb 12, 2024
models/migrations/v1_22/v287.go Outdated Show resolved Hide resolved
models/webhook/hooktask.go Outdated Show resolved Hide resolved
@silverwind silverwind added the type/refactoring Existing code has been cleaned up. There should be no new functionality. label Feb 13, 2024
@silverwind silverwind added the type/enhancement An improvement of existing functionality label Feb 13, 2024
Co-authored-by: silverwind <[email protected]>
@oliverpool
Copy link
Contributor Author

@jolheiser do you know if/when you will have some time to look at this PR? (especially since it should reduce the changes needed for #19307)

@oliverpool
Copy link
Contributor Author

@silverwind @KN4CK3R thank you for your initial comments. Do you have any other feedback?

Do you think that there is a chance for this PR to make it before 1.22 ? (cc @delvh)

@delvh
Copy link
Member

delvh commented Feb 26, 2024

If we merge it this week, then yes.

@silverwind
Copy link
Member

I'm not well versed around these parts but what is the deal with this PayloadVersion? Is it a breaking change in the webhook payload?

@lunny
Copy link
Member

lunny commented Feb 27, 2024

Could you have more description on the issue body about the plan? Why you add the new version column and how will the new version support customized webhooks in the future?

@oliverpool
Copy link
Contributor Author

@silverwind & @lunny I haved updated my initial description to emphasize:

Does it addresses your concerns?

@delvh
Copy link
Member

delvh commented Feb 27, 2024

What I haven't understood yet:
You introduce the new version, 2.
However, I don't see this version being set anywhere, except in the tests.
Did I miss something?

@oliverpool
Copy link
Contributor Author

The version 1 is set to the hooktask which were generated before the upgrade (set to 1 in the migration, as you suggested).
Either the hooktask was not sent before the upgrade or the user requested an old hooktask to be sent again.

So the "new" codebase should only create version 2 hooktasks and after a while, no more version 1 should be present in the database.

@delvh
Copy link
Member

delvh commented Feb 27, 2024

I think we are talking about different things:

I expected to see the version 2 being set somewhere (apart from the tests).
And I don't see it anywhere.

So the "new" codebase should only create version 2 hooktask

Yes, that's exactly what I thought.
However, is that the job of other PRs?
Because I expected that it is done here…

@oliverpool
Copy link
Contributor Author

oliverpool commented Feb 27, 2024

@delvh it is done upon hooktask creation in services/webhook/webhook.go:172

webhook_model.CreateHookTask(ctx, &webhook_model.HookTask{
		HookID:         w.ID,
		PayloadContent: string(payload),
		EventType:      event,
		PayloadVersion: 2,
	})

@delvh
Copy link
Member

delvh commented Feb 27, 2024

Now it makes sense why I don't find anything when I'm searching for [vV]ersion.*=.*2… 🤦

services/webhook/deliver.go Outdated Show resolved Hide resolved
services/webhook/matrix.go Show resolved Hide resolved
Copy link
Member

@jolheiser jolheiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good from initial look. I'll try and take some time to toy around with it more.

services/webhook/packagist.go Outdated Show resolved Hide resolved
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 27, 2024
@6543 6543 added this to the 1.22.0 milestone Mar 1, 2024
@oliverpool
Copy link
Contributor Author

As soon as there is willingness for this PR to be merged, I can solve the merge conflicts.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Mar 7, 2024
@6543
Copy link
Member

6543 commented Mar 7, 2024

As soon as there is willingness for this PR to be merged, I can solve the merge conflicts.

I guess that would be now

@techknowlogick
Copy link
Member

I brought this up on the TOC call, and how this PR would be great to get in, Im just wondering if it'll still include a raw copy of the body and headers that are sent to the endpoints? As having that information has been helpful to be able to copy/paste it into a curl to debug issues.

(Not blocking getting this in, I'm just asking to make sure no existing functionality is removed by this PR)

@jolheiser
Copy link
Member

Yes, that part appears to be largely unchanged https://github.com/go-gitea/gitea/pull/29145/files#diff-2402142c5412a93d0288dee23e4fa6d8d5b1e8a07b627714820a2344ff205125R161

@oliverpool
Copy link
Contributor Author

if it'll still include a raw copy of the body and headers that are sent to the endpoints

Yes it does, a little bit adapted, because the body must be saved as well (since the payloadBody is now the raw event):

	// Record delivery information.
	t.RequestInfo = &webhook_model.HookRequest{
		URL:        req.URL.String(),
		HTTPMethod: req.Method,
		Headers:    map[string]string{},
		Body:       string(body),
	}
	for k, vals := range req.Header {
		t.RequestInfo.Headers[k] = strings.Join(vals, ",")
	}

I have solved the merge conflicts.

@6543 6543 merged commit 26653b1 into go-gitea:main Mar 7, 2024
26 checks passed
zjjhot added a commit to zjjhot/gitea that referenced this pull request Mar 8, 2024
* giteaofficial/main:
  Store webhook event in database (go-gitea#29145)
  Fix bug hidden on CI and make ci failed if tests failure (go-gitea#29254)
@oliverpool oliverpool deleted the webhook_store_event branch March 8, 2024 05:42
zjjhot added a commit to zjjhot/gitea that referenced this pull request Mar 9, 2024
@lunny lunny mentioned this pull request Mar 9, 2024
zjjhot added a commit to zjjhot/gitea that referenced this pull request Mar 9, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. modifies/migrations size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/enhancement An improvement of existing functionality type/refactoring Existing code has been cleaned up. There should be no new functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants